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H ARDWARE REDUNDANCY AND 
FAULT TOLERANCE 


After reading this chapter and completing the exercises, 
you will be able to: 


+ Prepare for emergencies that can cause system downtime 
Manage the power supply to your Linux computer 

+ Check the integrity of your Linux file systems 

+ Understand how redundant disk systems can protect data 


i: the previous chapter you learned how to set up and manage user accounts 
and multiple file sytems in Linux, which included creating new user accounts 
and new file sytems.You also learned basc information about how to view and 
manage processes in Linux. 


In this chapter you learn how to protect the data on your Linux system from 
downtime. This involves planning for potential trouble before it occurs and 
understanding how to manage key aspects of a Linux computer to avoid down- 
time. As you will learn in this chapter, the power supply and hard disks are key 
potential causes of downtime. 


PREPARING FOR EMERGENCIES 


As you probably know by now, computer systems are not infallible For all the dependence 
society places on computers, they continue to break down, leading to crises for customers, 
managers, and system administrators T hese crises loom largest for system administrators, the 
people everyone else looks to for immediate guidance and resolution when an organization's 
computer systems malfunction.T hese are the times when a system administrator who has pre- 
pared for emergencies really shines. Although handling crises is part of the sytem administra- 
tor’sjob description, those who excel in this area are on the path to promotions or raises when 
the proverbial xy falls C onversely, those who have not prepared to handle difficult stuations 
are likely to be looking for work elsewhere. 
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Creating a Disaster Plan 


The first tep in preparing for emergencies is to do everything possble to avoid them. A 
“disaster” somewhere else doesn’t have to mean an emergency for your systems. R ather 
than sending out a distress call, how much better to send the following sort of message to 
management: “You may have heard that a tornado ripped through downtown Salt Lake 
City last night—1! just wanted you to know that our systems are all backed up, with full 
power, running smoothly and without interruption.” The preparation that allows such a 
message begins with a disaster plan— a written document stating how you have prepared 
your systems and what you will do in the event that various problems occur.A disaster plan 
should cover a wide range of potential problems Your plan may include guidelines regard- 
ing some of the following problems: 


a A hard disk crash. 
a An employee or intruder destroying data on the system or attempting to crash it. 
a A fire in the server room. 


a A natural disaster that wipes out all backups in the office, destroys the server, and 
blocks anyone from getting near the office. 


A disaster plan should outline policies and procedures to guide the actions of the sytem 
administrator and other staff members in preparing for and reacting to many types of emer- 
gencies. Information in a disaster plan should include the following: 


m Detailed sytem specifications for all hardware 
a A list of software installed on the server 


= Location of software masters, manuals, and licensing information for each 
application 


a Information about the server's power supply 


m Steps required to start the server using a boot or rescue disk (described later in 
this chapter) 


a Steps required to reinstall critical applications and data archives for those 
applications 


= Names and contact information for those to be informed of server emergencies 


= N ames and contact information for those who can be called to help in case of 
emergencies (including key technical support numbers for hardware and software 
vendors) 


The exact steps outlined in the plan depend on the nature of your organization. For exam- 
ple, a company that provides credit card services to customers around the world could not 
even allow a natural disaster that wiped out an entire office to top customers from accessing 
the network. In such a Stuation, millions of dollars are at stake; as a result millions should be 
gent in preparing systems that will prevent a problem in one area from blocking access to the 
entire sytem.A study done by O racle, the largest database software company, estimated that 
for its key customers, every hour of system downtime resulted in a loss of $80,000 to $300,000. 
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On the other hand, a mall company with a server or two isnot subject to such losses, though 
it may be just as dependent on its computer systems Small firms can take some ample pre- 
cautions that will greatly help in nearly all stuations A critical difference is the amount of 
time that the organization is willing to be without data. 


By performing all the taks on the following checklist, you could get any office up and run- 
ning again in 24 hours after most emergencies, at much less cost than keeping duplicate 
servers in cities around the country. 


= Keep a replacement hard disk and possibly other parts (power supply, network 
cards) in storage. 


m Back up the system each night to a rotating set of archive media (as described in 
Chapter 14). 


= Take copies of the back-up media off-site once per week (to another office a 
bank vault, or even someone's home in the case of a mall company). 


Asyou draft a disaster plan in consultation with management teams and others involved with 
the information systems at your organization, consider the monetary value of time Can you 
calculate the dollar value of each minute that data is inaccesible to employees or customers? 
M any companies can; they should then spend accordingly to keep systems up and running. 


Depending on your organization’s requirements, you should consder all of the following 
components as potential points of failure, setting up redundant systems when possible: 


m Hard diks 

=» Memory chips 

= Network cards 

= Power supplies (within a computer) 

a IncomingAC power (wall socket ) 

a Software failure (kernel or application crash) 


Preventing Downtime 


The term downtime refers to occasions when an organization’s computer systems cannot 
respond to requests for information. For example, aWeb server that doesn't respond to browser 
requests, a database server that can't respond to queries, or a print server that can't accept print 
jobs are all experiencing downtime. In computer parlance, these devices are down. It’s impor- 

tant to remember that a computer can be down without being powered off. If software or 
hardware problems prevent a computer from responding as intended, it is effectively down— 

because users can't perform tasks as they need to.T he rest of this chapter describes techniques 
that you can use to protect your Linux system from excessive downtime. Before you can suc- 
cessfully prevent downtime, you need to understand a few key concepts: 


=» Redundancy: A state of readiness in which a duplicate component can take 
over if the primary component fails A completely redundant system is a replicated 
system. M ost computer systems are not completely redundant. Because of cog 
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considerations, only those components that are most likely to fail or that cause 
the greatest problems if they fail are supported by redundant systems. 


= Fault tolerance: T he ability of a system to deal effectively with problems (also 
known as faults). Faults can include hardware failure, power failure, stolen data, 
and many others.T he more fault tolerant a system is, the more unexpected prob- 
lems it can handle without going down. 


= Points of failure: Weakness or vulnerabilities in the information sytem. 
Looking for the possble points of failure is a good way to strengthen the fault 
tolerance of your system.Work to eliminate any single point of failure— a sn- 
gle highly vulnerable part of a system whose failure will bring down the entire 
system. If a single (or most dangerous) point of failure cannot be eliminated 
entirely, try to make that part of the system redundant. 


Understanding High Availability 


Fault tolerant computer systems that run continuougy are said to provide high availability. 
A high availability computer system is able to serve data as much of the time as possble T he 
availability of a sytem may be measured as a percentage. For example, a sytem may have a 
gatigical availability (uptime) of 99.99%. M ost standard Pentium- class systems running Linux 
would probably have an availability of 95% or higher, depending on the environment in 
which they are running. (For example, if the municipal power supply was subject to frequent 
outages, the availability might be lower.) But improving system availability becomes increas- 
ingly difficult as you approach 100%.To achieve 100%, all posible causes of downtime mug 
be eliminated through redundant systems, elimination of points of failure, and so forth. 


It's important to keep in mind that while achieving 100% uptime is an admirable goal, it is 
only a theoretical goal. All systems will necessarily experience some degree of downtime.To 
understand the difficulty implied in attempting to achieve 100% availability, think of a graph 
in which a curve approaches a line (an asymptote). The curve of availability never actually 
touches the line of 100%, but as it gets closer and closer, the costs rise to infinity.T hus, com- 
panies are forced to set more practical availability goals, based on the amount of downtime 
the organization can realistically tolerate. O ne way a company can accommodate unavoid- 
able downtime is to schedule it in the form of maintenance or upgrades A network that is 
carefully maintained and upgraded in this way is far less likely to experience cody, unex- 
pected downtime. 


Downtime is generally measured as a percentage. For example, suppose your organization can 
tolerate one hour of downtime per year. In other words, out of the 8760 hours in a year, 1 
can be allocated to downtime (scheduled, if all goes well). Dividing 1 by 8760 gives a value 
of 0.0001142. M ultiplying by 100 gives the percentage of downtime: 0.01142%. Put another 
way, this organization requires 99.989% uptime. O nce you have calculated this goal, you can 
more easly determine the amount of time and resources required to achieve it. 


Within the realm of high availability systems, the term resource groups is used to refer to an 
application, the data that it requires to operate, and any other resources the application requires 
to complete its computing taks T he term high availability cluster is used to refer to a group 
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of servers that have high availability features and are dedicated to handling a common set of 
taks Each high availability cluster is devoted to handling a set of tasks or resource groups. 
Several methods are used to control how the servers in a cluster respond to problems T he two 
most widely used of these methods are: 


= One or more servers st idly waiting to take over a resource group (a task) in the 
event that a main server fails. W hen the main server is restored to activity, the 
back-up server becomes idle again as the main server takes over. T his method 
requires additional hardware equivalent to the main server to provide redundancy, 
which may be codly. 


m Each server in a cluster may handle various taks If one server goes down, the 
other servers take over the resource groups of that server.T his doesn't require 
costly hardware to at idly waiting for a failure, but performance may be seriously 
degraded if all servers are already busy when one fails. 


Linux supports some high availability features. T hese include U PS devices and both hardware 
and software R AID (these subjects are discussed later in this chapter). H owever, Linux does not 
support some high availability features, such as software control of high availability clusters and 
mainstream applications that are aware of high availability features. M any people in the Linux 
community are working on getting Linux to support additional software tools and hardware 
components. In the future these will allow Linux to compete better with UNIX vendors that 
have been working on high availability sytems for decades. T hese systems, from vendors like 
O racle, IBM , and H ewlett- Packard, may cog millions of dollarsT hey include all of the redun- 
dancy features described in this chapter, as well as others that Linux does not yet support. By 
increasing its high availability features, Linux can be used in more business critical computing 
environments. 


Creating Rescue Disks 


One of the most common system failures is a hard dik problem.T his can take the form of 
a complete crash, in which case you may have to restore all of the data to anew hard disk 
from a back-up archive. In other cases, part of the disk may be corrupted so that the sytem 
cannot boot correctly. Sometimes a hard disk problem can even be caused by something that 
the sytem administrator has done to the system. In the case of many hard drive failures, you 
cannot boot the sytem normally from the hard disk; you must boot the Linux system from 
a floppy disk before you can access the hard disk to try to solve the problem. T his disk is 
called a rescue disk. 


A rescue disk is a floppy disk that includes the files needed to run a minimal version of 
Linux without using the hard disks on your system. After booting the system from such a 
disk, you can mount the hard disks of your system (as described in Chapter 8) and edit files 
(including the LILO boot information) to correct problems A rescue dix is only one type 
of floppy disk you may use with your Linux system: 


a A rescue diX allows you to run Linux without relying on data stored on your 
system's hard disk. It contains a text editor such as vi, afew command-line utili- 
ties such aS 1s and cat, and a few utilities for managing hard disks such as 
fdisk.A rescue dix is used only for emergencies. 
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= A boot disk is used to launch the operating system stored on your hard dix. It is 
used in normal operating stuations to gart the Linux system. 


= An install disk is used when you first install Linux. Its contents are similar to a 
rescue disk, but it is not designed to asist you in recovering damaged data 


= A root disk is used in conjunction with an install disk during some Linux ingal- 
lations (depending on the Linux distribution you are using). 


Some system administrators always boot their servers from a boot disk.T his takes more time 
than booting from a hard drive, but because a server is rebooted only rarely, speed of boot- 
ing isnot really a concern. By using a boot disk, the sytem administrator can control how 
the system boots and easly maintain a backup of the boot dix in case of problems. A rescue 
disk is used when the boot dix or hard disk fails. 


Ideally, you would create a rescue dik for your sytem when installing Linux. T his guaran- 
tees that the rescue disk and the information on the computer's hard disk are compatible, 
which may help as you attempt to fix system problems. 


Asan example, the R ed Hat 6.0 Linux CD provides copies (or images) of the two disks 
needed to start the installation program: a boot disk and a root disk (the root disk contains 
a minimal root file sytem). O nce you create these disks, you can use them to start your 
R ed Hat Linux system at any time. T he images for these disks are located in the images / 
subdirectory of the installation CD.You can copy these images to a floppy disk using the 
rawrite utility in DOS or Windows (which is also on the R ed Hat CD). Alternatively, 
you can use the Linux dd utility by combining the dd command with the name of the 
image file (rescue, boot, Or root). For example: 


dd if=rescue.img of=/dev/fd0 bs=72k 


To start a Red Hat 6.0 system in emergencies, you must first boot the system using a 

boot disk, then insert the rescue disk to begin the process of dealing with problems on 
the system’s hard disks. Red Hat refers to this process as entering rescue mode. You can 
also create a boot disk in Red Hat 6.0 using the mkbootdisk command. 


Other Linux distributions require ïmilar techniques for creating a boot dik or rescue dix. 
M any installation programs also provide multiple virtual consoles that you can use to diag- 
nose or repair problems after booting the system from an installation disk. By switching from 
one virtual console to another, you may be able to mount ahard disk and diagnose or repair 
a problem. Press the Alt key and then a function key (F1 to F6) to switch between virtual 
consoles. W hen using an installation disk rather than a rescue dix for this task, keep in mind 
that the installation disk may not have the tools you need to diagnose and fix a problem. 


Maintaining Software M asters 


Asyou will learn in Chapter 14, most back-up strategies focus on backing up user data— that 
is, the information created by your organization, for use only within your organization. System 
administrators are often careless about backing up the software tools and operating system files 
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used to create and work with that data because they think these items can be purchased in any 
computer store. H owever, this is not always the case. C onsder the following problems that can 
occur after a system failure, when you attempt to reacquire your company’s software tools 


= The company that created the software may be out of business. N ote that this is 
mos likely to happen when the software is specific to one industry, such asa 
video store software package, chemical plant software, or medical office software. 


a The particular version used by the company may no longer be available from the 
vendor.A costly upgrade would then be required, with new training for all 
employees. 


m T he vendor may not be able to ship the replacement application software imme 
diately. For instance, if your system goes down on a Friday, the vendor probably 
wouldn't be able to ship the software until M onday, for delivery on Tuesday. T hus, 
the process of reconstructing the system data from backups could not begin until 
five days after the initial failure. 


In all of these situations, a sytem administrator would be better served if he or she had care- 
fully and securely stored the software masters (such as the installation C D ) originally pro- 
vided by the vendor. U nfortunately, software masters are often discarded along with the 
unread user manuals as soon as the products are installed. Instead, they should be stored along 
with backups of other system data and mentioned prominently in your disaster plan. W hen 
using complex applications such as client/server databases or customized business software, 
application data sometimes cannot be restored properly until the application is first installed. 
If you can immediately locate software masters for all important applications used on your 
server, you can then rebuild a system as follows: 


a R ebuild or repair hardware components such as hard disks. 
a Install the operating system and applications used by the company. 
m R estore user data files. 


If you can afford the additional capacity, it’s a good idea to back up the operating system 
tm and applications as well as user data. This safeguards your custom configurations as well 
as the applications themselves. But keep track of the software masters and documenta- 


tion nevertheless. You may need them to retrieve license information or activation keys 
for your applications. 


M ANAGING THE COMPUTER’S POWER SUPPLY 


The power supply is one of the easier components to protect within a Linux server. T his 
component converts the AC power from a wall socket to the low-voltage DC power used 
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by computer chips, disk drives, and other accessories. As shown in Figure 9-1, the power 
supply is usually the angle largest component insde your computer. It rarely fails if it is 
properly cooled by an internal fan or room air conditioner. For occasions when the power 
supply does fail, two possible remedies can be part of your disaster planning: 


Power supply 


Wall socket 
® 
A 
a >. A 
KAA ZA Power is distributed 
O to components within 


the system. 


Figure 9-1 The power supply component inside a computer 


m Keep a cond power supply on hand. If the power supply indde a computer fails, 
you should be able to replace the part and have the system running again within 
an hour. Spare power supplies are not overly expensive. 


= Purchase a server with a built-in back-up power supply.T his is milar to having 
an uninterruptible power supply (described in the next section) within the com- 
puter. It allows you to schedule a time to shut the server down when it is not 
being used so you can replace the failed power supply component. Servers with 
dual power supplies are generally quite expensive because of the control circuitry 
required to switch to the back-up supply in case of a failure. 


A power supply is a single component in your computer. Don’t ever try to 
disassemble and repair it, or you risk receiving a powerful electric shock. 
Caution Replace it as a complete unit. O pening a power supply unit is very danger- 


ous. A power supply contains components such as capacitors that can hold a 
large electric charge even when the power to the system is shut off. 


Connecting a UPS to a Linux System 


In most parts of the world, failure of or an unexpected irregularity in the AC power (the wall 
socket power) is much more likely than a failure of the computer's power supply. You can 
ameliorate the effects of irregular power by using devices known as surge suppressors. Surge 
suppressors provide a regular power sgnal to the computer's power supply. T hese devices 
remove voltage spikes, brownouts (moments of reduced voltage), and other irregularities (such 
as changes in the frequency of the sgnal— normally 60 cycles per second) that don’t affect your 
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refrigerator or office lights, but can damage senstive electronic equipment. Figure 9-2 illustrates 
how a surge suppressor removes irregularities so that the computer's power supply can rely on 
a consstent level of incoming power. 


Incoming AC power 
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Figure 9-2 A surge suppressor removes power irregularities 


W hen the power actually fails, however, a surge suppressor doesn't help. For these cases, an 
uninterruptible power supply (UPS) is needed.A U PS contains batteries that are contin- 
ually charged by the main AC power when it is active During a power outage, the batteries 
in the UPS take over, delivering power to the computer as if the outage had not occurred. 
Figure 9-3 hows a computer connected to a UPS. 


AC power UPS device 
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Figure 9-3 Computer connected to a UPS 


A UPS also includes surge suppressor features. Thus when the main AC power is active, 
Tm the power reaching the computer is filtered by the UPS; a separate surge suppressor is 


not necessary. 


Because batteries are expensve (and heavy), a low-cost U PS typically provides only 5 to 30 
minutes of battery power to a computer. Systems that require high availability may have a 
U PS costing thousands of dollars, that provides power for several days. Such high-end U PS 
systems may be integrated with gas- powered generators that recharge the U PS batteries. In 
most situations, however, a U PS is intended to provide power during momentary power 
outages, which generally last less than five minutes. In the event of a long-term outage due 
to weather or utility company problems, a U PS allows computer systems to be shut down 
gracefully.T his is especially important for a Linux computer system, in which data is cached 
in memory rather than being written immediately to hard disk. If the sytem is not shut 
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down properly, such data will be lost. W hat's more, when the Linux system is turned on 
again, the system may require a few minutes (or several hours) to check the integrity of the 
hard disk files and perform other checks.A U PS prevents these problems by giving the Linux 
system administrator time to shut down Linux properly, inform users that the server will be 
shut down, and exit system services without any loss of data. 


Automating Linux Shutdown 


Although aU PS allows time for a system administrator to shut down a Linux system grace- 
fully, the system administrator may not be present when the power outage occurs. For this 
reason, U PS devices typically have a serial communications port (Smilar to the connection 
used by a printer or external modem) that sgnals the Linux system in the event of a power 
outage. T he Linux system then executes ample scripts to determine what actions to take 
U PS devices can generally recognize three events, as described in Table 9-1. 


Table 9-1 Events Tracked by a UPS Device via a Serial Port 


Linux system response 


Power failure Begin a system shutdown | The shutdown process may take 5 to 
30 minutes, depending on the capacity 
of the UPS device and the type of 
work and number of users supported 
by the Linux server. 


Power is out and the Shut down quickly In this situation, Linux usually exe- 
UPS battery is low cutes an immediate halt command to 
write cached data to files and unmount 
file systems before the UPS battery 
fails. Users logged in to the system may 
have very little warning about closing 
their files, but typically no data is lost. 


Power has been restored | Cancel any pending When a nonemergency shutdown 

shutdown (taking 5 to 30 minutes) is in process, 
the shutdown can be canceled before 
any services are shut down. 


Event recognized by UPS 


Each item in Table 9-1 corresponds to asgnal number (Smilar to those used with the kill 
command, as described in Chapter 8). When a UPS is connected to a computer running 
Linux, the system administrator also installs a software daemon to monitor the serial cable 
that the U PS uses to communicate with the computer. Figure 9-4 shows a U PS connected 
to a Linux system via a serial cable. 
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Figure 9-4 Serial connection between UPS and Linux system 


M any different programs are available to monitor a U PS. O ne popular U PS manufacturer, 
American Power C onverson (APC), has Linux-specific programs designed to provide access 
to special control features of its UPS devices. Because mog UPS devices employ common 
communications systems, however, the standard Linux U PS daemon, known as powera, can be 
used with any U PS.T he powera software package may be included on your Linux CD-ROM. 
You can also acquire the powera software package from Linux FTP stes such as metalab unc edu 
in the /pub/linux/system/ups sSubdirectory.T he package on this FTP steisin .tar.gz 
format, so the command to prepare it on your system would look like this: 


tar xvzf powerd-2.0.tar.gz 


After installing the powerd software package, the system administrator must activate it by 
performing the following three steps 


1. Add powera to the start-up scripts so that it is tarted each time the Linux 
sytem boots. (T his step is described in the following section.) 


2. Set up the powerd configuration file.T his file controls how the powerd 
program monitors the serial communications line connected to your U PS. 


3. Check the /etc/inittab file to see what actions Linux will take based on the 
instructions sent from the powerd program. 


Autostarting the powerd Daemon 


Before the powerd daemon can start automatically each time you boot Linux, you must add 
it to the sytem initialization scripts described in this section.T he exact method required to 
accomplish this depends on your version of Linux, though the variations between Linux 
versions are small. T his section describes a typical Linux configuration. 


The powera software package includes a script used to start the powera daemon. T he 
powerd indallation process copies this script to the /etc/re.d/init.d directory. In 
some cases you may need to manually copy the script to the correct directory. For example, 
if you stored the powerd package in the /tmp directory, the command to copy the script 
would look like this 


cp /tmp/powerd-2.0/powerd /etc/rce.d/init.d/ 
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Once this script is installed in the correct directory, you can execute it (and start the powera 
daemon) with the following command: 


/etc/re.d/init.d/powerd start 
To gop the powera daemon, use this command: 
/etc/re.d/init.d/powerd stop 


T his powera dart-up script should also be added to your system initialization directories so 
that powerd darts automatically each time you boot your Linux system. 


To have Linux run the powera start-up script automatically when the system is booted, add 
a pointer from a run level directory to the powera dart-up script. 


The run level directories define which services Linux starts when using a given run level. 
R un levels 3 and 5 are most often used for standard systems. The run level directories are 
located in /etc/rc.d. For example, the directory used to initiate services for run level 3 is 
/etc/re.d/rc3.d. 


To add the powerd script to the run level 3 directory, create a link, or pointer, in that direc- 
tory that refers to the start-up script. Each link in this directory begins with a letter s followed 
by atwo-digit number.T he numbers indicate the order in which services will be started. 


The powerd daemon should be started before most other system services, so a value of 03 
is a good choice. To create this link, use the following command: 


ln -s /etc/re.d/init.d/powerd /etc/rce.d/rc3.d/S03powerd 


A amilar link (using a filename beginning with x for ki11) isused to indicate that the powera 
daemon should be shut down when run level 0 (halt) or run level 6 (reboot) is initiated by the 
sytem administrator. T hese two commands create links within the run level 0 and run level 6 
directories that point to the powerd script. Services are shut down in the reverse order of 
how they are started up, so the number used for these links is 90.T he powerd daemon will 
be shut down after almost all other system services are hut down.To create this link, use the 
following command: 


ln -s /etc/re.d/init.d/powerd /etc/rc.d/rc0.d/K90powerd 
ln -s /etc/re.d/init.d/powerd /etc/rce.d/rc6.d/K90powerd 


The exact numbers used on your Linux system, the directories where these files are con- 
tained, and the specific daemons started in each run level will vary depending on your 
Linux distribution. The numbers that you select are somewhat arbitrary. They should 
simply be between the numbers used for other services on the system. You can review 
these other services by examining the contents of the run level directories (such as 


/etc/re.d/rc3.d). 


Configuring the powerd Daemon 


The configuration file used by the powera daemon, named powerd.conf, is located in the 
/etc directory. By default, most of the optionsin thisfile begin with ahash mark (+), indicating 
that they are commented out (marked as comments and thus inactive). By reviewing the 
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comments, you can learn how each feature is used. T hen you can select the features you want 
to use and activate them by deleting the hash mark. Although the powera daemon supports 
many features, the configuration available to you depends on your U PS device.T he remainder 
of this section describes a generic configuration for Sgnaling a power failure. 


In order to be a successful sytem administrator, you don’t need to understand everything about 
serial Communications, but you will need to learn about how your U PS device sgnals a power 
failure or low battery condition. Specifically, to understand how the U PS informs the powerd 
daemon of a failure, you must understand the structure of aserial connection.A standard serial 
cable contains either 9 or 25 pins, as shown in Figure 9-5.T hese pins can be referred to by 
number, as indicated in the figure A mall number of these pins (wires indde the cable) are 
named for the functions they perform, such as carrying a bit of data or sgnaling that a bit of 
data was received successfully. (The exact names are abbreviations of the pin’s function. For 
example, the pin that indicates“ ready to send data’ is known by the label RTS.) 


9-pin serial connector (cable end) 


GND RxD 
DTR TxD 


DSR RTS 
CTS 


25-pin serial connector (cable end) 
GND CTS RxD 


Figure 9-5 Standard serial cable connections 


As one example, a Blackout Buster U PS from PK Electronics uses pin number 1 on a 9-pin 
cable to sgnal alow battery condition. W hen the U PS sets the voltage on pin number 8 to 
low, this indicates that the main AC power has failed. By contrast, anon-low voltage on that 
pin indicates that the power outlet is working. On any 9-pin serial cable, pins 1 and 8 are 
referred to, respectively, by the names DCD and CTS.T he /etc/powerd.conf file con- 
tains information about the pin numbers and corresponding names for 9-pin and 25-pin 
cables, but you must learn the specifications of your U PS to determine which pins or names 
to use in configuring the powera daemon.T he technical support number or Web ste of the 
U PS manufacturer generally has all of this information readily available. 
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UPS vendors’ software packages are usually designed to provide automatic shutdown 
for NetW are or Windows servers. Note that this software will not work with your Linux 
system. Very few UPS vendors currently support Linux. American Power Conversion 
(see http:// www.apcc.com) is one exception. 


The powerd program can be configured by making a few changes to the default configu- 
ration file. Within the /etc/powerd.conf file, update the following lines using a text edi- 
tor of your choice: 


1. Look for the line containing the word disabled.T his line indicates that the 


powerd program should not actively monitor the serial port. By default, the 
powerd configuration is disabled to prevent it from listening to a serial port 
before it has been configured properly.An improperly configured powerd dae- 
mon might listen to the wrong serial port and attempt to shut down the com- 
puter, even though the power supply is fine. If necessary, add a hash mark (#) to 
the beginning of this line to make it a comment. 


. Use the search feature of your text editor to locate the phrase debug yes in the 


powerd.conf file.T his tatement indicates that the powera program should not 
run in the background but should stay active on the command line after being 
garted. T his allows you to immediately end the powera program by pressing 
Ctrl+C as you test that powera works correctly. C hange the line debug yes to 
debug no.You may want to do this sep after testing the U PS configuration as 
described in the next section, but you should not reboot your system unless you 
have set this line to debug no. O therwise, your system may pause while waiting 
for a response on the serial port. 


. The section labeled serialline includes three lines of text that define how the 


powerd program will monitor the serial port. T he three lines define which serial 
port to monitor, the wire within the port to monitor, and the status of the line to 
look for (watch for either high voltage or low voltage). Uncomment these lines 
and set the correct values that you have determined by reviewing the documen- 
tation for your U PS device For example: 


serialline /dev/ttyS1 
monitor DCD 
failwhen low 


T hese values indicate that the second serial port (/dev/ttys1) should be 
monitored, that the DCD line within the port should be monitored, and that 
a failure agnal should be sent when the DCD line goes to low voltage. 


. Save your changes to the /etc/powerd.conf file and restart the powera 


daemon using the following two commands % that the configuration file is 
reloaded by powerd: 


/etc/re.d/init.d/powerd stop 
/etc/re.d/init.d/powerd start 
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You may see device names (such as /dev/cua0 or /dev/cuai1) within the 

powerd.conf configuration file. Formerly, the serial port devices on a Linux computer 
were known by the names /dev/cua0 for the first port, /dev/cual1 for the second port, 
and so forth. The preferred names for serial port devices in Linux have been changed to 
/dev/ttyso for the first port, /dev/ttyS1 for the second port, and so forth. You may 
still see the /dev/cuax device names—and they will still work on most Linux systems— 
but try to use the newer /dev/ttySx device names where you have a choice. 


Although a UPS has only one serial communications cable attached to it, every U PS has 
several power outlets, thus several Linux servers can be plugged into one U PS. Each Linux 
server that is powered by an outlet on the U PS should run a copy of powera, even though 
the serial cable isnot connected to each of the Linux servers.T hanks to additional features 
of the powera daemon, it can use a network connection to inform a copy of powera 
running on other Linux servers of a power outage. T hus, powera on those other servers 
can watch for a message on the network from another copy of powerd.W hen this message 
is received, each copy of powerd can shut down the server on which it is running. The 
remoteserver linein the /etc/powerd.conf file isused to configure this remote mon- 
itoring feature. Figure 9-6 shows how several servers can be plugged into one U PS device. A 
§ngle serial cable allows the U PS to inform one server of a power outage T hat server then 
informs other servers via the network. (All of these messages are communicated between 
copies of the powerd daemon.) 


J Network 
All 3 servers l backbone 
are powered -A 
by the UPS. p | 
Server C 


UPS device a 


| Serial cable ae = Server A informs 


wa B f UP 
Only Server A is informed Server A peal oe i 
of UPS/main AC power network 
status via serial cable. : 


Figure 9-6 Multiple servers connected to one UPS with remote server shutdown 


Responding to a Power Outage 


The powera daemon doem't hut down your Linux system. Instead, it watches the serial 
port for information from the UPS device. W hen it receives information indicating the 
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power is out, the U PS battery is low, or the power has been restored, it communicates a signal 
to the init program.The init program is a control process that starts the first processes 
on Linux, such as the login screens. 


The init program is configured by a file called /etc/inittab. When the powerd 
daemon sends a signal to the init daemon, init checks the /etc/inittab configura 
tion file to determine what action to take. The file /etc/inittab normally includes two 
or possibly three lines indicating how Linux should repond when a power outage occurs, 
the U PS battery is low, or the power is restored. T he exact content of these lines (that is, the 
exact actions they specify) depends on your Linux vendor and version, but these lines taken 
from R ed Hat 6.0 Linux are good examples. 


pf::powerfail:/sbin/shutdown -f -h +2 "Power Failure; System 
Shutting Down" 


pr:12345:powerokwait:/sbin/shutdown -c "Power Restored; Shutdown 
Cancelled" 


T he first part of the line (p£ or pr in these two example lines) isa label chosen by R ed H at. 
The second part, powerfail and powerokwait, indicates to the init daemon what is 
being configured: either power failure or power restored. O nly two possible signals are rep- 
resented here. (Testing for a low battery condition is not part of the /etc/inittab 
configuration in R ed Hat 6.0.) In another distribution, the configuration line for a low 
battery would look something like this: 


pl:powerfailnow:/sbin/shutdown —h now "Battery Low..." 


All three of these sgnals make use of the shutdown command. In each of the three rele- 
vant lines in /etc/inittab (monitoring sgnals for power failure, U PS battery low, and 
power restored), different parameters are included after the shutdown command to deter- 
mine how shutdown operates. Each of the three commands is described in the following 
list. In each case, the message in quotation marks is part of the shutdown command and 
appears immediately on the screen of every user who is logged in to the Linux system. 


m shutdown -f -h +2 "Power Failure; System Shutting Down" 
Initiates a timed shutdown that warns all users on the system that the server will 
be halted (shut down and not automatically rebooted, using the -h option) in 
two minutes. N o system services are actually shut down until the end of the two 
minutes. If your U PS can supply power for a longer time, you could change the 
+2 parameter to another number— for example, to +10 for 10 minutes. 


m shutdown -c "Power Restored; Shutdown Cancelled": Cancels an 
existing shutdown.T his command is useful after a timed shutdown has been 
announced to users but before the time has expired. If power is restored, there is 
no need to shut down the system, so another announcement is sent to all users 
that the shutdown has been canceled and they can continue working normally. 
T his would be used if a short power outage causes the init program to 
announce a shutdown in 2 to 10 minutes (or more, depending on your U PS), but 
the power was restored after a hort time 
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m shutdown -h now "Battery Low": Initiates an immediate shutdown (halt 
and do not automatically reboot, using the -h parameter with now), disconnect- 
ing any users that are connected via the network and ending all applications. 

W hen the U PS battery is failing, the time left before the power is cut to the com- 
puter cannot be determined, so this rather impolite command is used.T hough it is 
unpleasant for users because it ends their Linux sesson without warning, it is better 
to preserve the file sytem integrity than to risk losing data because the battery fails 
before a hutdown has been completed. As a rule, you can avoid low battery condi- 
tions by careful planning. Ideally, your U PS device would be adequate for your 
Linux system, which means this command would never be necessary. 


You can update the information in the /etc/inittab file % that the init daemon exe- 
cutes any command that meets the needs of your system.T he shutdown command is effec- 
tive for unmounting file sytems and preventing data loss, but you might also want to create 
a script (see Chapter 12) that completes a set of tasks related to shutting down the system. 
T hese tasks might include: 


a Writing an incident to a log file 

a Sending an e-mail message to another system 9 
m Signaling a back-up server to take over a critical task 

a Writing information to a networked hard dix so it can still be accessed 

a Starting the shutdown command to halt the system 


CHECKING FILE SYSTEM INTEGRITY 


M aintaining the integrity of file systems is a key part of protecting data on your Linux sys 
tems. In order to maintain the file system, it's useful to understand some basics about the 
information stored on a file sytem and the arrangement of that information.T he definitions 
in the following paragraphs apply to all Linux ext2 file sytems. 


T he superblock is a set of information about the entire file sytem. It includes items such 
as the following: 


=» Maximum number of times the file system can be mounted between complete 
file sytem checks (using the fsck utility described in the next section) 


a Size of the file system in blocks (the standard block size is 1024 bytes) 
= Number of free blocks on the file system 


= Number of reserved blocks (reserved blocks can only be used by the root user or 
another user or group that the root user specifies) 


=» Amount of time snce a complete integrity check was performed on the file sytem 


If the superblock is corrupted, the entire file sytem is unusable. C opies of the superblock are 
gored in multiple locations across the hard dis during the formatting of an ext2 file sytem 
to protect against a superblock problem rendering the sytem unusable. Some parameters tored 
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in the superblock, such as the maximum number of times the file sytem can be mounted 
between integrity checks, can be configured during formatting or reset usng the tune2fs 
utility described later, in “Tuning a Linux File System.” Some features of the file system, such 
as the sze of a data block, cannot be changed without reformatting the entire file sytem. 


An inode (pronounced eye-node) isa file information record.A file record contains a file- 
name and the inode that holds information needed to access the file The file record is an 
indirect pointer to the file’s data: it points to an inode, which in turn points to the datain a 
file. A directory record isa file containing a lig of files and corresponding inode numbers 
used to access data in those files. 


Each inode contains all of the information about a file except the file's name. T his informa- 
tion includes the following: 


a The file access permissions 

= The owner and group of the file (noted as a user ID and group ID number) 

= The time and date of the last access, last modification, and file creation 

= Number of blocks used by the file on dix and the precise dze of the file in bytes 
m A pointer to the blocks of data in which the file's contents are tored 


Because of the way inodes are arranged on an ext2 file system, a file sytem can have only 
a limited number of inodes and thus a limited number of files. T he number is large, and 
you're never likely to run out of inodes for creating new files But you will notice as you use 
the fsck utility in the next section that the utility displays messages indicating the maxi- 
mum number of files that the file system can hold. For example, the utility output reading 
43686/247808 files indicates that of 247,808 posible files on this 1 GB hard dix, 
43,686 have been created thus far. 


T he brief explanation of inodes provided in the previous paragraph should help you under- 
stand how links are used in the Linux file system.A link allows two or more file records to 
refer to the same physical data stored in a file sytem. Among other things, a link makes it 
possible for two users to have two different filenames listed in their home directories, but 
actually be working with the same data when they edit those two files Links are of two 
types: symbolic and hard. A symbolic link is a file record that includes a path and file- 
name, but not an inode W hen a user refers to asymbolic link, Linux looks at the path and 
filename mentioned in the file record— sort of like an internal UR L pointing to another 
filename.T he file record for that second path and filename includes an inode.T hat inode 
is used to access the file data for the symbolic link. 


A hard link isa file record that includes a filename and inode, just like a regular file record. 
But a hard link refers to an inode that already has a file record pointing to it.T he hard link 
is a second file record pointing directly to the same physical data. A single inode can have 
numerous file records (hard links) pointing to it. After a hard link is created, it is equal to the 
first file record that points to the same inode If the first file record is deleted, the second file 
record (created as a hard link) is unaffected— it still refers to the same file data. 
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Both symbolic and hard links are used often in a Linux file system. W henever you use the 
1s -1 command, any symbolic links are indicated as extra filenames in the right column of 
the output. An example of a symbolic link in R ed H at Linux is the view command. T his 
output shows how you can use the 1s -1 command to show the view command: 


$ ls -1 /bin/view 
lrwxrwxrwXx 1 root root 2 Aug 12 13:36 view -> vi 


T he arrow in the right column indicates that the file named view is a symbolic link to the 
file named vi (located in the same directory, because no pathname is included).T he letter 1 
in the far left column of the screen output also indicates that the file is a symbolic link. 


T he number in the second column from the left (in this case, 1) indicates the number of file 
records that refer to the same inode as this file record. 


The zcat filename (also in R ed H at Linux) is an example of a hard link. The zcat file 
record refers to an inode that two other file records also refer to. The 1s -1 command 
again shows this: 


$ ls -1 /bin/zcat 
-rwx-r-xr-x 3 root root 50384 Mar 25 13:28 zcat 


In this sample screen output, the number 3 in the second column from the left indicates that 
the file record holding the filename zcat refers to an inode that two other files also refer to 
(for a total of three). Figure 9-7 shows the difference between a symbolic link and a hard 
link. In this figure, a programming language named Perl is stored in afile called per15.003, 
where the 5.003 indicates a precise verson number. O ther file records also point to the 
same information (inode) using different filenames. T his allows users who might not know 
the precise verson number to access the Perl programming language file. 


Symbolic link file record 


Points E link_to_perl 
another file record = 
File perl 5.003 


record 


A Points to an Hard link file record 
an Inode existing inode [pet | 

Inode with ID number, Gee 

file structure, 
and access info 


Points to 
data blocks Hard disk or 
in the file system other file system device 


Figure 9-7 Hard link vs. symbolic link 
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To see the inode numbers associated with the filenames in a directory, use the -i option of 
the 1s command. O nly the first few lines of the output for this directory are shown in this 
listing: 

$ ls -i /bin 

22549 arch 

22497 ash 

22498 ash.static 

22507 awk 

22514 basename 

22475 bash 

22499 bsh 

22495 cat 

22477 chgrp 

22478 chmod 

22479 chown 

22502 consolechars 

22480 cp 

22504 cpio 


Using the fsck Utility 


You can run the fsck utility at any time to check the integrity of a Linux file sytem.T he 
fsck utility can quickly determine whether any file system problems are evident. If none 
are apparent, fsck runs very quickly.You can, however, force fsck to execute a full suite of 
file system tests even when the utility would otherwise exit quickly because no errors are 
apparent.T his is called forcing the fsck utility. 


Each time you boot Linux, the fsck program runs automatically to check all mounted file 
systems. U nder normal circumstances, this automatic check takes only a few seconds Two 
gtuations will cause the fsck utility to spend more time checking the file systems: 


a Linux was not shut down gracefully (all sytem services were not shut down in an 
orderly way). Data was therefore not written correctly or fully to the file system. 


m The file sytem has been mounted numerous times After a certain number of 
mount operations, Linux forces the £sck utility to go through a complete check 
of the file system even if no problems are evident. 


W hen the second condition occurs, Linux displays a message like the following as you boot 
the sytem: 


Maximal mount count reached: check forced. 


Just as Linux occasonally does a thorough (forced) file system check automatically, you can 
also use the fsck utility to manually check each of your Linux file systems Such a com- 
plete check is wise before any disk- intensive operations, including the following: 


= Backing up the system completely before a hardware upgrade 
m Defragmenting the file system (as described in the next section) 
m Attempting to repair data in a corrupt (damaged) file system 
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To force a complete check of a file sytem, use the fsck command with the -£ option, fol- 
lowed by the name of the device on which the file sytem is located.T he following example 
of the £sck command checks the file sytem located on the first partition of the second IDE 
hard disk: 


You can only use the £sck command on file systems that are not mounted. 
A | Using it on a mounted file system will damage the file system. 
Caution 


fsck -f /dev/hdbl 


You can also use the mount point in the directory structure For example, if the first parti- 
tion of the second IDE hard dik (/dev/hdb1) is mounted as /database, you could also 
use this command: 


fsck -f /database 
M ultiple file systems can be checked at the same time by listing them. For example: 
fsck -f /dev/hdb2 /dev/hdb3 /dev/hdcl 


The £sck program runs all the file sytem checks at the same time, querying you as ques- 
tions arise about file sytem information. If a file sytem you are checking is fine, you see 
only a brief output after running fsck, as follows 


Parallelizing fsck version 1.14 (9-Jan-1999) 
e2fsck 1.14, 9-Jan-1999 for EXT2 FS 0.5b, 95/08/09 
/dev/hdb2: clean, 43686/247808 files, 906478/987997 blocks. 


Launching fsck with the -£ option provides more detailed information. Sample output is 
shown here: 


Parallelizing fsck version 1.14 (9-Jan-1999) 

e2fsck 1.14, 9-Jan-1999 for EXT2 FS 0.5b, 95/08/09 

Pass 1: Checking inodes, blocks and sizes 

Pass 2: Checking directory structure 

Pass 3: Checking directory connectivity 

Pass 4: Checking reference counts 

Pass 5: Checking group summary information 

/dev/hdb2: clean, 43686/247808 files (0.3% non-contiguous), 
906478/987997 blocks. 


N ote that running a complete file system check using the -£ option takes several minutes. 
A 1 GB Linux file sytem that is almost full of data may take three minutes to check. Larger 
file systems take considerably longer.W hat's more, using fsck to perform file sytem repairs 
on a Linux system with multiple or large file systems may take up to an hour to restart the 
Linux system.You can avoid time-consuming file system checks by shutting down Linux 
properly every time. 
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ungraceful shutdown does not allow Linux to store information to files, stop system 


A proper system shutdown in Linux is often referred to as a “graceful shutdown.” An 
services, and cleanly unmount file systems. 


The screen output of the fsck utility shown above contains five complete passes through 
the file sytem. The fsck utility will automatically correct any problems detected during 
each of the five passes. If an automatic correction is not possible, a query will appear on 
screen. T hese queries are smple questions aking you whether to save or delete unidentified 
pieces of information, or to provide some other piece of information that the file sytem 
requires. It is rare to have fsck ask a question during a file system check unless a serious 
disk problem occurred, such as the power to the Linux system being cut off during a 
complex disk operation. 


W hen Linux is booted, the fsck utility isrun with the -a option, which automatically cor- 
rects any minor errors.T hese minor errors might include such things as a missing timestamp 
on a file if the sytem was not shut down properly. 


In some cases it may be necessary to indicate to the fsck utility the type of file system being 
checked. T his is done with the -t option. For example: 


fsck -t ext2 -f /dev/hdb2 


Defragmenting a File System 


All file systems are divided into units of storage called blocks A standard block contains 
1024 characters (bytes). All of the files stored in a file system are broken into pieces and 
stored in these file sytem blocks. O n many operating systems, the information stored in a 
file sytem becomes fragmented.T hat is, the blocks that make up individual files are read 
all over the surface of a hard dix or other device. R eading an entire file on a fragmented 
hard disk requires much more time and movement of the hard disk mechanism than read- 
ing nonfragmented files, which are stored in sequential sections of the hard disk.W indows 
NT isparticularly noted for the need for frequent dix defragmenting. (D efragmenting is 
also called disk optimization because it optimizes access time to files and reduces wear 
on the hard disk.) 


Fortunately, the ext2 file system used by Linux is not subject to much file fragmentation. 
W hen new files are written to a Linux hard dix, the files are tored at locations spread across 
the entire hard disk. T his allows space for each file to grow in size without becoming frag- 
mented. In contrast, most file systems used by other operating systems begin storing a new 
file at the first available location on the hard disk. A s new files are added, they fill up the space 
immediately next to existing files As a result, if an existing file increases in sze, the operating 
system cannot smply squeeze more information into a space adjacent to the file. Instead, it 
mug allocate the necessary space from a different part of the hard dix. As more and more files 
become broken into piecesin this way, the disk becomes fragmented. Figure 9-8 illustrates the 
difference between an ext2 file sytem and most other file systems in this regard. 
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Figure 9-8 Creation of new files in ext2 and in other file systems 


W hen you run the fsck utility (or watch it run automatically as Linux is booted), the screen 
displays the percentage of files on each file system that are fragmented. T his percentage is 
normally less than 3%. 


H ard disks that are becoming very full, however, are likely to experience more file fragmen- 
tation as pace for new or expanded files becomes scarce. In this case, it can be helpful to 
defragment your Linux hard disk. Because this operation is rarely done in Linux, the utility 
used to defragment a Linux hard dix is not included with most Linux products You can 
download it from the Internet usng an FTP ste such as mæalab unc edu. M ultiple utilities are 
provided in the defragment package to allow you to work with various types of file systems. 
The utility used to defragment a standard Linux ext2 partition is called e2defrag. 


When you need to defragment a Linux file system, begin by unmounting the file 

system. Then be certain to run fsck to check the hard disk’s integrity. To complete 
these tasks on the root file system, you must boot Linux from a floppy disk so the 
root partition on the hard disk is not mounted. 
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Tuning a Linux File System 


The dumpe2fs command provides many pages of information about a Linux ext2 file sys 
tem.T his command can be used on any Linux file sytem, whether mounted or not mounted. 
It provides details about the options set in the superblock of the file system and information 
about the data blocks A paging program such as less should be used to view the output of 
dumpe2fs, asit runs to many pages A sample command is hown here: 


# /sbin/dumpe2fs /dev/hda3 | less 


You can use the tune2fs utility to specify how an ext2 file sytem acts in certain stua- 
tions.W hen you format an ext2 file sytem, the mke2£s command described in Chapter 8 
uses default options for the items described in this section.You can add parameters to the 
mke2f£s command to set those options as you format a new file sytem, or you can use the 
tune2fs command after the file system is formatted. M ost of these options are stored as 
part of the superblock on the file system. 


The tune2fs command should never be used on a file system that is 
| mounted. Unmount the file system that you need to adjust, booting Linux 
Caution 


from a floppy if the root file system must be tuned. 


T he tunable parameters for an ext2 file sytem affect the file sytem’s stability and, to a lesser 
degree, its performance. Some configuration options of a file sytem, such as the block sze, 
cannot be changed without reformatting the file sytem and using a parameter on the 
mke2fs command. The primary options available with the tune2fs command are 
described in the following paragraphs. 


You can set the maximum number of times the file system can be mounted without having 
a complete integrity check using the fsck utility. A lower number will check for errors 
more often, but will also sow down system startup more often as the system is being 
checked. You can use the dumpe2fs command to see what the current maximum mount 
count value is set to.A sample command to set this value is shown here: 


# /sbin/tune2fs -c 10 /dev/hda3 


T he number of times the file system has been mounted since the last time it was checked is 
recorded in the superblock.You can change this value using the -c option. By lowering or 
raising this number in relation to the maximum mount count parameter, you can control 
when the next forced automatic integrity check will occur. 


You can determine the Linux system's response when a file system error is detected. T he 
response is normally continue. H owever, you can choos to stop the Linux kernel when 
an error is detected, or remount the file system as read-only instead of read/ write. (N ote that 
mounting the file system as read-only may make the operating system unusable, although 
safe for examination and repairs) A sample command is shown here: 


# /sbin/tune2fs -e remount-ro /dev/hda3 


Part of each Linux file system is set aside as reserved space. T his pace cannot be used by any 
regular sytem users for normal file storage. Instead, it is generally only available to the root 
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user when the hard disk becomes full so that root can continue to operate the system in 
order to free space for other users. If the entire root file sytem (including this reserved 
pace) becomes full, the system is basically unusable. 


T his reserved space on a file system is normally 5% of the total file system space. H owever, 
you can configure this amount as a percentage of the total file system size or as a number 
of data blocks on the file system. You configure these options using the -m or -r option 
with the tune2£s command.T he root user can always access this reserved space, but you 
can also allow another user ID number or group ID number to access it by using the -g 
and -u options of tune2fs.T his might be useful, for example, if you have a file system 
dedicated to a database application and wish to reserve some space for the database 
administrator's group. 


In addition to the dumpe2fs command, you can use the -1 option of tune2fs to lig the 
contents of the superblock. Because the tune2fs command sets options within the super- 
block, viewing the current settings with the -1 option can be helpful. 


UNDERSTANDING REDUNDANT DISK SYSTEMS 


One of the mos vulnerable parts of a computer system is the hard dis. It contains many 
moving parts, and it may be subjected to constant heavy us. Its health is crucial to the health 
of the entire system and to users’ productivity, because the hard disk contains the data that the 
computer sytem was designed to provide. For these reasons, a great deal of effort has gone 
into making hard disks redundant— so they don’t form a potential single point of failure in a 
system. Although redundant hard disks have long been available on mainframe systems cog- 
ing millions of dollars, most users now rely on groups of hard disks, known as redundant 
arrays of inexpensive disks M ore commonly, these groups of diss are called RAID wub- 
systems or R AID arrays. 


T he idea behind R AID is this rather than attempting to create a sngle expensive hard disk 
that never fails (a technological impossbility), it makes more sense to use a group, or array, 
of standard (inexpensive) hard disks. T he assumption is that if one disk fails, the others can 
take over its duties until the failed disk is replaced. T he safety of aR AID system liesin the 
extreme improbability of all the disks in the array failing at the same time. O f course, this 
assumes that high-quality components are used— an important assumption for R AID. 


The management or control of aR AID system can be implemented in hardware or soft- 
ware. It can contain as many disks as necessary to reach the storage capacity needed.A R AID 
system does not have a specific number of hard disks, a specific storage capacity, or even a 
specific platform. RAID systems are used by all operating systems. Instead, the different 
forms, or levels, of R AID are distinguished by the techniques used to store data, as explained 
in the following sections. 
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Defining RAID Levels 


R AID can be implemented in many forms, or levels.T he levels differ in the amount of fault 
tolerance they provide and the cost of implementation.T he next sections introduce the dif- 
ferent R AID levels and define the terms and techniques associated with R AID. 


When reviewing RAID technologies or products, look for specific features rather than 


Vendors occasionally differ in the features they associate with the different RAID levels. 
just a level number. 


RAID-Linear 


RAID-Linear is a technique that makes it posible to combine multiple physical devices into 
a sngle logical device.T his allows one logical file sytem (as mounted in Linux) to an mul- 
tiple disk drives or partitions T his is useful when you want a file system to be larger than a 
gngle hard disk. For example, suppose you need a huge file system for the /home directory. 
By storing this file sytem on multiple disk drives, you avoid buying one very large (and very 
expensve) hard disk. Instead, you can purchase multiple diks of a more common sze. But 
the /home directory can dill be managed as a single file sytem because of R AID - Linear. 


By itself, this system doesn't provide any fault tolerance or data protection. R AID -Linear is 
not a true RAID level because it does not provide any fault tolerance, nor does it improve 
system performance as some R AID levels do. In fact, R AID-Linear reduces fault tolerance 
because if any one of the dis in a R AID -Linear array fails, the entire file system is unus- 
able. R AID -Linear is illustrated in Figure 9-9. 


RAID-0 (Striping) 


RAID-O makes use of a data storage technique known as striping. W ith striping, a sngle 
piece of data is divided into pieces and tored on more than one hard disk. T his technique 
allows the system to access data faster, because two disk drives work together to gather parts 
of any requested information at the same time. T he performance gain is increased if each 
hard disk uses a separate hard dix controller, such as having two SCSI cards; the multiple 
hard dix controllers prevent a bottleneck in communicating with the CPU viaasingle data 
channel. H owever, if either hard disk failsin aR AID -0 setup, the entire file sytem is unus- 
able. H ence, using R AID-0 without other measures described in this chapter reduces fault 
tolerance sgnificantly. Figure 9-10 illustrates the use of striping. 
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Without RAID-Linear 
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Figure 9-9 RAID-Linear 
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Figure 9-10 RAID-O, or disk striping 
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RAID-1 (Disk Mirroring and Duplexing) 


RAID-1 mirrors data across multiple hard dixs T he term mirroring is used to describe a 
system with two or more hard disks that contain identical information. Each time one hard 
disk is updated, the copy or copies are also updated. If one hard dix fails, the mirrored hard 
disks continue to respond to data requests without interruption. Duplexing is identical to 
mirroring except that the hard disks are on separate controller cards T he separate controller 
cards provide increased performance and reduced vulnerability to a hardware failure 
Duplexed hard disks are sometimes referred to casually as mirrored disks, though in fact the 
use of multiple controller cards makes the two techniques distinct. D uplexing is the preferred 
technique when possible. 


Mirrored and duplexed hard disks provide both increased performance (because multiple 
hard disks respond to data requests at the same time) and increased fault tolerance (because 
if one hard disk fails, the duplicate disks continue to work without interruption). M irroring 
or duplexing disks does have two downsides. First, the time required to write files to a hard 
disk is increased because data must be written to each disk instead of jug one. Second, these 
techniques require extra hard disks that are used only for mirroring—they don’t provide 
extra torage capacity. For example, if you want to mirror a 10 GB hard dix, you must pay 
for 20 GB of hard dix space.W ith the low price of hard disks today, this is generally a very 
good investment, though it can be a limiting factor when planning a system that includes 
large amounts of storage (hundreds or thousands of GB of data). Figure 9-11 shows a mir- 
rored hard disk system. 


Identical data 


is written to 
disks A and B 
We hold these truths We hold these truths 
to be self-evident to be self-evident 
Disk A Disk B 


V 
One logical device 
/dev/md0 


Figure 9-11 RAID-1 with mirrored hard disks 


RAID-3 (Striping with Parity) 


RAID-0 (striping) improves performance but makes a system more vulnerable to failure. 
RAID-3 combines the performance advantages of striping data across multiple hard disks but 
provides additional protection against the failure of one of the hard diksin the form of parity. 
The term parity refers to a technique that allows corrupted data to be reconstructed usng 
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an extra piece of information that is created as the data is tored.T he parity information pro- 
vides redundancy to the piece of data In R AID -3, this extra information regarding how files 
are divided up is stored in a parity stripe. If one of the hard dixs fails, the system can use 
the parity information to reconstruct the data stored on that disk. Figure 9-12 illustrates a 
R AID-3 system. 
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Figure 9-12 RAID-3, striping with parity 


R AID-3 provides good fault tolerance because of the parity stripe. If a hard disk fails, data is 
gill usable; when the disk is replaced, the data is rebuilt without any intervention by the ys- 
tem administrator. R AID -3 also improves performance on systems that read a lot from the 
disk.All the hard diksin aR AID -3 array can work in parallel to respond to multiple requests 
at the same time. Instead of waiting for a single dix, the chances are good that a hard disk 
that is not busy can immediately begin to service a read request. Also, unlike dix mirroring, 
with R AID-3 only one hard disk (the one containing the parity information) is unavailable 
for data storage. For example, to have 80 GB of storage space in aR AID-3 system, you only 
need to purchase 90 GB of hard diks The write performance of RAID-3 does suffer 
because the parity information must be computed and several hard disks must store infor- 
mation for each write operation. 


in production systems and so are not presented here. These two RAID levels simply use 


Although RAID-2 and RAID-4 are listed in some definitions of RAID, they are not used 
different combinations of the technologies implemented by other RAID levels. 


RAID-5 (Striping and Parity) 


RAID-5 is similar to R AID-3 except that with R AID-5 both the parity information 
and the stored data are striped across multiple hard disks.T his has the advantage of mak- 
ing read performance better, but it makes write performance even worse than R AID-3. 
As with RAID-3, if a hard disk fails, the parity information allows the information to 
be reconstructed once the disk is replaced. In the meantime, data remains available. 
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Figure 9-13 illustrates how RAID-5 spreads information across multiple hard disks. 
M any vendors who sell R AID-5 hardware systems (see the next section) use built-in 
write caching to store new information in memory until it can be written to the mul- 
tiple hard disks without degrading performance overall. 
File A contents 
We hold these truths SS 


File B contents Sarre 
to be self-evident Sas 


File C contents Se 
that all men are 


created equal Disk A Disk B Disk C 


One logical device 
/dev/md0 


Figure 9-13 RAID-5, parity and striping over multiple disks 


Using Hardware-Based RAID 


A RAID system can be implemented as a separate device that connects to your Linux 
computer like an external printer or CD-ROM drive. T his type of RAID is known as 
hardware-based RAID because the control and management of the disk array relies on 
a hardware system (called aRAID subsystem) that is separate from the main computer. 
H ardware- based R AID devices include separate memory, a CPU, and other components 
besdes the array of hard dis. A hardware-based R AID system contains a separate micro- 
processor and control circuits; it is practically a separate computer with the sole purpose of 
providing data to your main system.T he main advantage of using a hardware-based R AID 
system is that all of the special technology is contained in the R AID device.T he operating 
system itself cannot differentiate between aR AID device and any other hard dix.T his elim- 
inates the need for any special software or configuration on the Linux system. 


Another advantage of hardware R AID systems is that they can provide features that are 
rarely duplicated without a separate device to house the hard disk array.T he most promi- 
nent of these features is hot-swapping, which makes it possible to pull out and replace a 
failed hard disk without ever turning the system off.T he hot-swapped hard disk is gradu- 
ally reintegrated into the array as data is written to it; as a result, the failed hard disk entails 
no downtime.T his feature obviously adds to the expense of aR AID system. 


and that are designed to manage or configure the RAID device. You must be certain 
that such utilities can be used on a Linux platform. See the Linux HOWTO document 
“DPT-Hardware-RAID HOWTO,” which explains how to use hardware RAID devices 
on Linux. The end of the next section describes where to find HOWTO documents. 


A hardware RAID system may include utilities that run on the host operating system 
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T he main disadvantage of hardware-based R AID is the cog. Because the R AID system has 
a complete suite of electronics to manage the hard disks, plus a separate case and software 
utilities, hardware-based RAID is much more expensive than the software-based R AID 
described next. 


Using Software-Based RAID 


The term software-based RAID refers to R AID that is controlled and managed by soft- 
ware on the computer system that uses the hard disk array.A software-based R AID array 
includes only the hard dixs the CPU and memory of the computer system to which the 
array is attached control how the disk array is used.T he software used to control this type of 
RAID array consists of the operating system (the Linux kernel) and additional special device 
drivers or file system management tools. Because those who use Linux systems are often 
concerned about the high cost of proprietary computing hardware, software- based R AID is 
an attractive way for Linux users to achieve the fault tolerance and performance improve- 
ments associated with R AID. 


To implement software-based RAID on Linux, you can purchase a commercial product 
designed to support Linux. O ne such program is theTwinC om duplexed hard disk product 
(see www.twincom.cm). Commercial products have the advantage of being fairly easy to 
install; in addition, the vendors usually provide technical support either for configuration or 
problem solving. 


It is not necessary to choose a commercial verdon of software-based R AID. T he features 
built into the Linux kernel for support of software-based R AID devices are less expensve 
and more flexible. H owever, mog system administrators choose not to use these features 
because they are fairly new and difficult to use. Because these features add fault tolerance 
without excessive cox (only the extra hard disks and the sytem administrator's time), you 
should familiarize yourself with them. 


R AID support in the Linux kernel is called Software-R AID. Linux support for softwarebased 
R AID usesa set of kernel modules called multiple device drivers, which are identified in the 
/ dev directory by the designation md. After enabling the multiple device drivers in the Linux 
kernel, you can use the RAID software utilities (called raidtools) to configure multiple 
hard disks in accordance with one of the R AID levels described previougy. R AID -Linear, 
R AID-0 (sriping), R AID-1 (mirroring/ duplexing), and R AID-5 (striping with parity) are 
supported in the Linux verdon 2.2 kernels (This is the kernel shipped with Linux versons 


350 


Chapter9 Hardware Redundancy and Fault Tolerance 


such as R ed Hat 6.1, SuSE 6.2, and O penLinux 2.3.) Figure 9-14 shows the screen from 
which you can configure R AID options 


A complete discussion of setting up R AID using these kernel features is beyond the scope of 
this book, but the raidtools package and copious additional documentation are available 
to guide the sytem administrator in this taskk.You'll find a great deal of helpful information 
in the mini-HOWTO called“ Software-R AID” (though named “mini,” thisH OWTO islong 
and thorough). R egarding the specialized tak of uding RAID for the root partition of a 
Linux system, see the HOWTO document called “R oot-R AID.” These documents are 
available at many Internet stes where the Linux HOWTO documents are stored, such as 
ftp:/ / metalab unc edu/ pub/ L inux/ doc/ H OW TO /.M ox Linux distributions also include a copy 
of the HOWTO documents. For example, R ed H at 6.1 includes these files on the first 
CD-ROM in their package at /doc/HOWTO and /doc/HOWTO/mini. 
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Figure 9-14 Screen used to configure Software-RAID 


REDUNDANT SERVERS 


The first part of this chapter mentioned using duplicate servers located in different cities as 
a method of providing a highly fault tolerant information sytem. H owever, Linux does not 
yet have the ability to accommodate redundant servers in the same way that some operat- 
ing sytems can.A redundant server requires very high peed data communications lines and 
special software to both mirror information and inform each system of when the other is 
responding to requests and when it should respond because of an outage of some sort. 
Figure 9-15 shows how two redundant servers might be connected after additional hard- 
ware and software support is added to Linux. T he connections include: 


a A serial cable over which a brief repeating sgnal (known as a heartbeat) informs 
each system of the status of the other systtem.T his Sgnal allows special software 
on each server to know when to take over functions because the other server is 
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down. (In some server configurations, a high-speed network connection is used 
instead of a serial cable.) 


a A hardware RAID subsystem with a“twin-tailed” cable T he twin-tailed cable 
allows one RAID subsystem to be connected to two computers Both computers 
can access information Smultaneoudgy. If either server goes down, the other server 
gill has complete access to the same data. 


= Dual network cards in each server, so that if one network card were to fail, the 
server could still communicate with clients on the network. 
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Figure 9-15 Redundant servers 


Although standard Linux does not yet support true redundant servers, you can use several 
technologies to improve the fault tolerance of your Linux systems. C onsder the following 
ideas for improved fault tolerance of Linux systems in large organizations 


= You can us a commercial product such as theTwinC om Linux dik mirroring 
software mentioned previoudy to mirror hard disks across a network link. T hough 
this doesn't create a true redundant server that can automatically take over if the 
other server fails, it does replicate data in two different locations. 


= By configuring your router and Domain N ame Service appropriately, you can 
ensure that two or more computers are available to respond to requests for Web 
pages, FT P information, or other Internet services Special routers are available to 
gread incoming requests among several servers that contain identical information. 
T hus if one server goes down, another server will respond to the incoming 
requests T his arrangement isn't appropriate for a mall system used by only a few 
people (such as a departmental database server), but it is very helpful (though 
expendve) for Web stes that cannot tolerate downtime. By using a product like the 
network-based mirroring in the previous item, you can postion these different 
servers in different offices or cities. 
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= Some Linux products support a set of multiple servers working in tandem to 
run Web servers or other network services that require constant uptime. (O ne 
notable vendor of such products isTurbo Linux; see the company’sWeb ste at 
www.turbolinux.com.) T hese additions to a gandard Linux distribution manage the 
transfer of control between systems to handle large numbers of incoming requests. 
Figure 9-16 illustrates how a DNS name server can route incoming requests for a 
Web page among a set of Web servers. T his setup allows the set of Web servers to 
handle a large traffic volume with redundancy among the Web servers. 
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DNS server == multiple identical 
== Web servers 


= 
Each Web server could 
also use separate 
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Web server Web server Web server Web server 


RAID subsystem 


Figure 9-16 A set of Web servers responding to Web page requests 


m The high-end server systems sold by the major hardware vendors (such as IBM , 
Toshiba, Compaq, Dell, and Acer) now support Linux.T hese servers support fea- 
tures like hot-swap hard disk bays for use with R AID software, redundant power 
Supplies and cooling fans, and other fault tolerant hardware features. 


You may have heard of a Linux development called Beowulf. This technology allows 
multiple Linux-based computers to be tied together in a parallel- processing architecture 
to simulate a supercomputer for work requiring massive amounts of computation (such 
as calculating weather patterns or world economic models). Beowulf is not intended to 
create redundancy for a single server, however, because each server in a Beowulf clus- 


ter takes on a specific portion of the computing task. 
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CHAPTER SUMMARY 


o Maintaining high availability of a Linux system is increasingly important for many orga 
nizations. A disaster plan and an understanding of the hardware and software options 
available can help reduce downtime for a Linux system. Creating a disk from which a 
failed system can be booted and maintaining a software master for all server applications 
are two easy steps that help a system administrator to prepare for possible emergencies. 


o The power supply to a computer is a key point of possble failure. By using surge sup- 
pressors, maintaining back-up power supply parts, and installing a U PS, the power to a 
Linux system can be safeguarded. 


a Hard disks are the mog vulnerable failure point in most computer systems. By checking 
file sytems regularly and using redundant systems such as R AID, information on hard 
disks can be protected against costly system failures. R AID systems can be controlled by 
separate hardware devices or directly through software support in the Linux kernel. 
Several levels of R AID are defined, according to how information is arranged on a set 
of hard disks Different R AID levels provide various performance, reliability, and cost 
trade-offs. 


KEY TERMS 

AC power — T he standard alternating current power coming into a building from a 
public utility; the power from a wall socket. 

array — A collection of multiple hard dixs See R AID. 

Beowulf — A cluster of multiple Linux servers operating in parallel as a supercomputer to 
solve complex problems. 

block — A unit of storage on a file sytem.A standard block contains 1024 characters (bytes). 

boot disk — A dix used to launch the Linux operating system stored on your hard disk. 
It can be used in normal operating situations to start the Linux system. 

defragmenting — The process of rearranging the files on a file system so that all the 
parts of a file are located next to each other on the physical hard disk. Defragmenting a 
hard disk increases system performance and reduces wear on the storage device. 

directory record — A file containing the names and inode numbers of other files. 

disaster plan — An organized written plan that describes how to respond to various 
threats to an information system such as Linux. 

disk optimization — See defragmenting. 

downtime — Occasions when an organization's computer systems cannot respond to 
requests for information. 

dumpe2fs — A utility used to display technical statistics and parameters about a Linux 
ext2 file system. 

duplexing — Term used to describe a system in which the contents of two file systems, 
that are located on different hard disk controllers, contain identical information. 
Compare to “mirroring,” a technique that provides identical information on two file 
systems but without redundant disk controllers. 
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e2defrag — The utility used to defragment a standard Linux ext2 partition. 

fault tolerance — The condition of being able to tolerate errors or events that might 
otherwise cause system failure. 

file record — An information item within a ext2 file system that includes a filename 
and an inode number.T he inode itself contains detailed information about the file. 

fsck — A utility used to check the integrity of an ext2 file sytem. 

hard link — A pointer to an inode that is already pointed to by at leat one other file record. 

hardware-based RAID — A RAID array that is contained in a separate hardware 
device (aR AID subsystem) and is controlled by aC PU and other components separate 
from the CPU of the Linux system. 

high availability — Term used to refer to the processes, products, or programs involved 
in ensuring that a system experiences as little downtime as possble. T he goal for all 
high availability systems is 100% uptime 

high availability cluster — A group of servers that process the same tasks (resource 
groups) and take over each others functionality in the event of an outage or failure. 

hot- swapping — T he process of removing and replacing a failed hard drive from a R AID 
hardware device or specialized server without turning off the power to the device. 

init — A daemon that acts as a control process to start the first processes on Linux, such 
as the login screens. 

inode — A file information record, identified by a unique number within a file system, 
which contains detailed information about a block of data commonly called a file 

install disk — A disk used when you first install Linux. Its contents are amilar to a rescue 
disk, but it is not designed to asig you in recovering damaged data. 

link — A special file record that refers to the same physical file data as another file record. 

mirroring — Term used to describe a system in which the contents of two file systems 
contain identical information. M irroring improves data access peed and provides fault 
tolerance in the event that one of the file systems fails. 

parity — A technique that allows corrupted data to be reconstructed using an extra piece 
of information (the parity information) that is created as the data is tored. Parity infor- 
mation provides redundancy for a piece of information. 

parity stripe — Parity information stored as part of aR AID-3 or RAID-5 system. 

points of failure — Parts of an information sytem that are subject to failure. 

power supply — The component within a computer system that converts the incoming 
AC power from a wall socket or U PS device to the correct voltage for use by compo- 
nents in a computer. 

RAID — A system using multiple inexpensive hard disks arranged in a predefined pattern 
(an array) to improve performance, increase fault tolerance, or both. 

RAID-0— A RAID level that uses striping to improve dik performance but without 
adding any fault tolerance. 

RAID-1— A RAID level that uses dik mirroring to sgnificantly improve fault toler- 
ance. Disk read performance is also improved, but disk write performance suffers. 
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RAID-3— A RAID level that uses Striping with parity information to improve both 
performance and fault tolerance. 

RAID-5— A RAID level in which data striping with parity is spread across all disks in 
the RAID array (compared to R AID -3, in which the parity information is tored on a 
gngle hard dix). 

RAID-Linear — A method of combining multiple physical devices into asingle 
logical device. 

RAID subsystem — A hardware-controlled R AID device containing aC PU and other 
components to control the array of hard disks. 

redundancy — Term used to refer to a duplicate system component or piece of data. 

M any fault tolerant systems rely on the use of redundant components or data; in the 
event of a failure, the duplicate component or copy of the data would gill be available 
redundant arrays of inexpensive disks — SeeR AID. 

rescue disk — A disk created specifically to boot a Linux system in the event of a sytem 
failure. C ontains the software tools most likely to be of help in diagnosing and repairing 
the problem with the failed system. 

rescue mode — A mode of operation in R ed Hat Linux that is initiated by starting the 
system using a rescue disk. R escue mode is used to repair a system failure that blocks 
normal booting and operation. 

resource groups — T he taks and their accompanying system resources that are defined 
within a high availability cluster. Each server in the cluster can take over a complete 
resource group if the server handling that resource group fails. 

root disk — A dix used in conjunction with an install disk during some Linux installa- 
tions (depending on the Linux distribution you are using). 

single point of failure — A system component which, if it alone fails, renders a system 
unusable. C an be a hardware component or a piece of data. 

software masters — Original copies of an application supplied by a software vendor or 
manufacturer, usually one or more CDs, tapes, or disks. 

software-based RAID — A RAID array that is controlled or managed by software on 
the computer system that uses the hard disk array (the Linux system) rather than by a 
separate CPU or other hardware components. 

striping — A technique in which parts of a file are written to more than one dix in 
order to improve performance. See R AID-3. 

superblock — T he master information record for a Linux file system. 

surge suppressor — A device that prevents potentially damaging electrical irregularities 
from reaching a computer system's power supply. 

symbolic link — A file record that includes a path and filename, but not an inode. 

tune2fs — A utility used to view or adjust parameters within the superblock of a Linux 
file sytem. 
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uninterruptible power supply (UPS) — A device capable of providing power to a 


computer via batteries when the incoming AC power (wall socket power) fails. It also 
informs the Linux system of the status of the power. 


write caching — A technique in which information to be written to a file system (par- 


ticularly aR AID file sytem) is tored in sytem memory temporarily in order to 
improve performance of reading and writing information to the file sytem. 


REVIEW QUESTIONS 


1. A good disaster plan should consider versus benefit. 


a RAID level 
b. dangers 

c. cot 

d. availability 


. How does the amount of time that a company can be without data affect the contents 
of a disaster plan? 


. Why isit important to maintain software masters rather than rely on vendor support? 


4. At a minimum, a rescue disk should contain which of the following: 


a. A virtual console option 

b. A text editor and dix repair utilities 
c. A Beowulf configuration 

d. The file system superblock 


. Points of failure in a system have a direct relation to the of 
the system. 


a. fault tolerance 
b. redundancy 
c. cot 

d. write-caching 


. A built-in back-up power supply is the same thing as an uninterruptible power supply. 
True or False? 


. A surge suppressor protects against all of the following except: 
a. Power outages 
b. Voltage spikes 
c. R educed voltage 
d. Brownouts 


. What program is used to monitor aU PS and send a signal indicating that Linux 
should shut down? 


10. 


11. 


12. 


13. 


14. 


15. 
16. 
17. 


18. 
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. A shutdown command can be canceled if the main AC power is restored. True 


or False? 

T he configuration file for the powerd daemon includes the following: 
a. Scripts used when shutting down Linux 

b. The serial port and serial pin to monitor 

c. Debug information used to correct system flaws 

d. The call to the shutdown command 

The init daemon does which of the following: 

a. M anages all processes and receives signals from the powerd daemon 
b. Monitors the U PS via a serial cable 

c. Manages the powerd.conf configuration file 

d. Creates a high availability server 

W hich condition is normally not monitored by aU PS or the powerd daemon? 
a. Main AC power failure 

b. UPS battery low 

c. Main AC power restored 

d. Telephone power failure 

R edundant arrays of inexpengve disks (R AID) are used to provide: 

a. Lower-cost systems than single disks 

b. R edundant superblock information 

c. Fault tolerance and improved performance 

d. Beowulf clusters 

Striping refers to which of the following? 

a. Spreading a single file across multiple hard disks 

b. Duplicating file information on multiple hard disks 

c. Adding error- correcting codes to a file 

d. Duplicating inode data 

Define the advantages of using hot- swapped disk drives. 

Linux does not yet support any high availability features. True or False? 


A high availability cluster is a Beowulf cluster used for parallel processing. True 
or False? 


Software-based R AID support in Linux is configured as part of which of the following? 
a. Both symbolic and hard links in the file system 

b. The write-caching configuration file 

c. The Linux kernel and related modules and drivers 

d. The fsck and tune2fs utilities 
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19. 


20. 


21. 


22. 


23. 


24. 


25. 


If data contained in a file system’s superblock is unreadable, the file system cannot be 
accessed. True or False? 


An inode contains information about a file, including all of the following except: 
a. The file's name 

b. The file’s location on the hard disk or other device 

c. The file’s inode number 

d. Dates and times of creation and last access 


Symbolic links refer to inode numbers, hard links refer to paths and filenames. True 
or False? 


Describe the purpose of reserving a percentage of the space within a file sytem for 
the root or other user account. 


At boot time, the fsck utility does the following: 

a. Automatically checks each mounted ext2 file system for errors 
b. Unmounts any damaged file systems 

c. R eads and resets key superblock parameters 

d. Calls the tune2¢fs utility to manage file sytem start-up 


T he maximum number of times a file system can be mounted between complete 
fsck integrity checks can be configured usng tune2 fs. True or False? 


Disk fragmentation is not a problem in Linux ext2 file systems because: 

a. N ew files are created at locations spread around the entire file system. 

b. Striping used by the Linux Software-R AID feature prevents fragmentation. 
c. The fsck utility defragments the file system each time the system is booted. 
d. Fragmentation is not a sngle point of failure for a high availability system. 


HANDS-ON PROJECTS 


Project 9-1 


eet} In this activity you create a floppy disk containing an ext2 file sytem and work with that 
file sytem using the fsck and tune2fs commands T his allows you to experiment with 
concepts explained in this chapter without damaging the data on a hard dixk.To complete 
this project you need a blank disk and an installed Linux system. 


1. 
2. 
3. 


Insert a blank 3%-inch disk in the disk drive of your Linux computer. 

Log in to Linux aS root and open a command-line window. 

Format the dix, creating an ext2 file system, using this command: /sbin/mke2fs 
/dev/£d0. Study the information that appears on screen during the formatting 


process to see what items you recognize from the discussion in the chapter (such as 
inodes, blocks, and the superbock). 
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4. Dump the information from the floppy file system using this command: 
/sbin/dumpe2fs /dev/f£d0.T he listing is hort because the floppy is a small file 
system. R eview the information found on the superblock to see which fields you 
recognize from the chapter discussion. (O ther fields are described in the dumpe2fs 
manual page.) 


5. Change the maximum mount count for the floppy file system using this command: 
/sbin/tune2fs -c 5 /dev/fdo. (The default value when you created the file 
system in Step 3 was 20.) 

6. Check that the new value for maximum mount count is stored on the disk by using 
the -1 option of the tune2£s command to view the superblock (this is smilar to 
the dumpe2f£s command but only dumps the superblock parameters): 
/sbin/tune2fs -1 /dev/fd0. 


7. Check the integrity of the floppy file system using this command: fsck /dev/fdo0. 
T he results appear within a second or two because the file system is clean. 


8. Mount the floppy disk so you can write a file to it, as follows: mount -t ext2 
/dev/£d0 /mnt/floppy. (The /mnt/floppy directory exists on most Linux 
systems. You can create this directory if necessary, or use a different mount point if 9 
you wish.) 

9. Copy a system file to the floppy dix using this command: cp /etc/termcap 
/mnt/floppy.As the file is being written, eject the dix. (A large file is referred to in 
this example command to give you time to eject the dix while the copy operation is 
gill in progress.) T his simulates an error condition or power outage that may damage 
the file system. 

10. R einsert the floppy disk. 

11. Unmount the floppy disk using this command: umount /dev/fdo0. 

12. Run the fsck command again as follows fsck /dev/£d0.T his time the command 
shows the message /dev/fd0 was not cleanly unmounted, check 
forced. Depending on what was in process at the moment you ejected the disk, 
fsck May ask you a question about the disk data, or it may automatically update and 
correct any damaged status information. 


13. M ount the file system again asfollows mount -t ext2 /dev/fd0 /mnt/floppy. 

14. View the file system contents to see if you copied the file successfully, using the 
following command: 1s -1 /mnt/floppy. Because the copy operation was 
interrupted, the file is probably not there. 

15. Unmount the file system again and eject the disk using this command: umount 
/dev/£d0. 


Project 9-2 


In this activity you review information on the root file system.T hisis amilar to the previ- 
ous activity except that you work directly on your hard disk rather than on a blank floppy 
disk. Follow the instructions carefully to avoid damaging your Linux hard disk.To complete 
this project you need an installed Linux system. 
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1. Log in to Linux as root and open a command-line window. 
2. Enter the mount command with no options to see which device contains your root 


file sytem.T he first line of the output from the mount command includes this infor- 
mation.T he line looks like the following sample output except that the first part (the 
device name) will be different on your system: 


/dev/hda3 on / type ext2 (rw) 


. Use the dumpe2fs command to review information about your root file sytem. 


Pipe the results through the Less command % you can page through them one screen 
at a time. T he necessary command looks like dumpe2fs /dev/hda3 | less (sub- 
gitute the device name for your root file sytem where /dev/hda3 iS used). 


. Look for the line containing File system volume name in the output of the 


dumpe2fs command. Is a volume name defined on your sytem? 


. Look for the line containing Mount count in the output of the dumpe2fs com- 


mand.W hat does this number represent? H ow does this number relate to the line 
labeled Maximum mount count? 


. Use the tune2fs command to lit information from the superblock of your root 


file sytem. U se this command, substituting the device name of your root file system: 
tune2fs -l /dev/hda3.H ow does this information compare to the first part of 
the output from the dumpe2£s command? 


. The tune2£s command cannot be used to change parameters on a file system that is 


mounted. Because the root file sytem is aways mounted when you boot your sys 
tem normally, you must boot from a disk (such as a rescue disk) to change parameters 
on your root file sytem.View the man page of the tune2fs command to review 
what parameters can be changed using this command. Can you identify a hard disk 
parameter that cannot be changed with tune2fs? 


Project 9-3 


In this activity you review the Software R AID HOWTO document. T his document con- 
tains additional technical information about usng R AID on Linux.To complete this project 
you need an Internet connection and a workingWeb browser. 


1. 
2. 


Start your Web browser and go to the Linux HQ ste at: www. linuxhq. œm. 


Click the link for the Linux Docy Info section. (You can also explore other sections 
of this Ste to learn more about Linux kernel development.) 


. On the Linux Information page, click the link for HOWTOs Additional documenta- 


tion links are provided for many Linux- related subjects. 


. On the Linux HOWTO Index page, click 3.2 mini-HOWTOsT he mini- 


HOWTOs cover more focused topics they are not necessarily smaller documents. 


. When the long list of mini-HOWTO documents appears, scroll down and click 


Software-RAID mini-HOWTO. (You may al% find many other interesting topics 
to study on the lit of mini- HOWTOs) 
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6. R eview the sections of the Software R AID mini-H OWTO document. Click on a sec- 
tion, read or Kim some of it, then try the navigation links at the top or bottom of the 
section to switch to the previous or next section, or back to the table of contents for the 
Software-R AID mini- HOW TO.W hat warnings have you noticed about usng R AID? 


7. In section 1 of the mini-H OW TO, find the list of related Linux R AID documents. 
Select one that looks interesting and follow it to read more about R AID. 


CASE PROJECTS 


1. You have been asked to provide consulting services as an expert system administrator 
to Thompson M utual Funds, a nationwide financial services company. T hompson 
employs customer service representatives who respond to customer inquiries about 
their accounts, via telephone, 24 hours per day.T hompson also maintains aWeb site 
with information about mutual funds and access to stock market pricing information 
and individual accounts (where customers can view their balance, transfer funds 
between accounts, and so forth).T hompson has two main offices, one in San 
Francisco and another in Miami.All of the customer data is tored on Linux-based 
database servers. W ithout listing specific hardware and software requirements, what key 
measures would you use to try to protect Thompson's computer systems and provide 
high availability? D escribe in general terms the features of your disaster plan, what it 
protects against, what risks it does not fully address, and how costly each part of the 
plan might be to implement. 


2. Consdering the strong financial postion of Thompson M utual Funds and the lost value 
caused by any sytem downtime, discuss which R AID level or levels you might use in 
setting up the storage systems on a Linux server within a ingle T hompson office. 


3. TheThompson Web server is located in the San Francisco office Because existing 
customers use the Web server to access account information, Thompson wants to 
avoid any downtime for the Web server.Web server uptime is not as important as 
keeping the main database servers running, because customers can always call a cus- 
tomer service representative if the Web ste is down. Describe the relationship between 
the Web servers and the database servers. (Is one dependent on the other?) W hat high 
availability features would be appropriate for the Web server? 


